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I. INTRODUCTION 



A. PURPOSE OF THIS STUDY 

The purpose of this study is to examine two methods of statistical sampling 
which may have application in assisting government price analysts and contract 
negotiators in expediting processing of proposals for change orders. The study will 
describe how and why the use of statistical sampling may expedite proposal negotiation 
while maintaining acceptable levels of risk, and will determine which of the two 
sampling methods examined provides more acceptable results under the given 
conditions. 

B. NEED FOR THIS STUDY 

Current defense acquisition procedures often involve situations in which 
Department of Defense (DOD) agencies must deal with a sole source supplier in 
buying material or services. In major weapon system acquisitions for example, the 
Department of Defense typically issues a large number of change orders to modify an 
existing contract. A lead ship or aircraft production contract may generate over 10,000 
change orders. Why must such a large volume of change orders be issued? After a 
prime contract is awarded and production begins, design changes are often necessitated 
by a change in performance requirements requested by the government or by unforseen 
technical problems which almost always seem to crop up. Each design change requires 
a modification to the prime contract called a change order. In each case, the 
contractor prepares a proposal reflecting his estimate of what the requested change will 
cost. The two parties (government and contractor) must then negotiate a price for 
each change; and, because the prime contractor is the logical one to incorporate the 
requested change, there is no competition to help assure that the government receives 
the fairest possible price. The oniy mechanisms working to assure a fair price for the 
change order are the adequacy of the contractor's estimating procedures, the 
contractor's inherent honesty and desire to provide a good product at a fair price, and 
the analysis of the proposal by government price analysts. 

Federal Acquisition Regulations (FAR) require the government to analyze each 
proposal prior to negotiation to assure that the proposal represents a fair price. The 
analysis and negotiation of costs for each proposal is done by some cognizant 
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government agency. Often, a group of government employees have been assigned to 
perform such functions in residence at the contractor's plant. The volume of work 
thus generated and the amount of money involved are quite substantial. This volume 
of work combined with a lack of sufficient numbers of government analysts leads to 
large backlogs of unprocessed proposals. To perform a really thorough analysis and 
patient negotiation takes much more time than government analysts are currently able 
to give to a proposal. If analysts do try to take more time and be more thorough, they 
fall still farther behind as the backlog continues to grow. Therefore, there is 
tremendous pressure on analysts to expedite their work even though it is generally 
recognized that hurried analysis and negotiation can result in costly overpayment since 
quickness commonly works against thoroughness and accuracy [Ref. 1: pg. 2]. 

Unprocessed proposals can result in extra expense for the contractor too. In 
many situations involving ongoing production or repair work, the proposed work is 
begun before the proposal is analyzed and negotiated to avoid expensive delay and 
disruption costs. The contractor, however, except for partial advances called "progress 
payments", is not paid until after the proposal is processed. Because of this, the 
contractor may have to borrow working capital to cover funds tied up in the backlog 
thus suffering capital costs. 

It is generally recognized that it is in the best interest of both parties to expedite 
the processing of the proposals without sacrificing accuracy. If allowed by the 
regulations, analyzing and negotiating a sample of proposals selected with an effective 
statistically-based sampling technique could have these effects. If a suitable sample of 
proposals were selected from the backlog and carefully analyzed and negotiated, the 
resulting data could be extrapolated to estimate what the results would have been had 
ever}' proposal received the same treatment. 

The reason for auditing the proposal population is to ensure the proposals refect 
costs that are fair and reasonable. During an audit the government analyst will find 
that one of three possible conditions exists. First, the government may feel that the 
proposal has been understated, i.e. , the contractor's proposed cost for a change to the 
contract is less than the actual cost the contractor will incur. Second, the government 
may conclude that the contractor's proposal is overstated. Third, the audit findings 
may conclude that the proposal is reasonable. Any overstatement or understatement is 
considered to be an error in the proposal population. A sampling technique which 
allowed no sampling error (with sampling error being defined as the chance that a 
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sample which is statistically selected and evaluated will lead to the wrong conclusion or 
to an inaccurate projection) would select a sample from the population of proposals 
which, when audited, would always give an estimated value for the population as a 
w'hole which was exactly correct, no matter what the degree or distribution of errors in 
the proposal population. Since sampling errors are due entirely to chance and are 
inherent in any sampling process, we cannot expect a sample of "n" proposals from the 
proposal population to provide an error-free characterization of the "N" proposals in 
the population. 

Assuming, then, that there will be some degree of error in the prediction of the 
true value of the entire proposal population whenever a sampling technique is used; the 
behavior of the degree of error must be predictable and exhibit certain qualities in 
order for the sampling technique to be considered appropriate for the purpose 
described above. 

Specifically, the degree of error should not be easily altered by the distribution, 
size, or type (overstatement understatement) of errors found in the population. If 
certain patterns of errors caused the entire population to be evaluated as understated 
then the government would pay more than a fair price for the changes described by the 
population of proposals which contained those errors [Ref. 2: pg. 7], Since both parties 
to the negotiation would have to enter into a binding agreement to abide by the results 
of using statistical methods, all aspects of the sampling and estimation process must be 
disclosed in advance. With this necessary advanced knowledge, a shrewd contractor 
could carefully seed his proposal population with deliberate errors of the appropriate 
size, type, and distribution and thereby be awarded a larger payment from the 
government. 

Therefore, the desired sampling technique will not necessarily be the one which 
results in the most accurate, average estimate for various error arrangements in the 
proposal population. It will instead be the method which responds least to variations 
in the arrangement of errors. 

C. METHODOLOGY 

As mentioned previously, the purpose of this study is to examine the effectiveness 
of two sampling techniques that appear to be most suitable for the purpose of 
analyzing contract change orders. The two sampling techniques to be studied are 
Stratified Random Sampling and the Basket Method. In this study, the two methods 
will be used to draw samples from populations for evaluation. The populations were 
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previously used in a joint study of the American Institute of Certified Public 
Accountants and the American Statistical Association [Ref. 3]. The data consist of two 
columns of values which represent the proposed, or book value of a contract change 
and the audited or true value of the change. The populations are rigged with either 
random or planned errors. The samples drawn by the two methods from each 
population will be evaluated and compared to determine which method gives a better 
estimate of the whole population according to the goals described above. Beth the 
error rigging and evaluation steps are explained further in the description of the 
simulation. 

The amount of work associated with auditing is more closely correlated to the 
number of items being audited than to the total dollar value of all the items being 
audited. Therefore, the sampling rules of the two methods will be adjusted so that they 
will draw samples with the same number of proposals from each population. The 
results of this study will then indicate which method yields the more desirable 
prediction while holding the cost of the audit constant. 



10 



II. THE BASKET METHOD 



A. HISTORY 

The "Basket Method" of sample selection was developed by Dr. K. T. Wallenius. 
Professor of Mathematical Sciences at Clemson University. Development of the 
Basket Sampling method was sponsored by the Office of Naval Research and Naval 
Material Command and funded by the Office of Naval Research under its Acquisition 
Research program. The Basket Method was developed as a potential tool to assist 
price analysts and contract negotiators in expediting processing of proposals for change 
orders when dealing with a sole source supplier. 

B. DESCRIPTION 

The name "Basket Method" is derived from the manner in which the population 
is partitioned into separate groups (baskets) prior to randomly selecting one of the 
baskets as the sample. The goal of partitioning the population into baskets by the 
basket assignment process is to make each basket a good representation of the 
population as a whole. It must be stressed at this point that "representative" should be 
thought of in terms of bid prices only. 1 Because each basket is representative of the 
population as a whole, the spread and proportion of proposal values will be nearly 
identical to those of the population. Therefore, it makes no difference which basket is 
selected to be audited in detail. The following example will describe the use of the 
basket method technique. 

1. Basket Assignment 

Imagine having a population of 100 proposals (N = 100) from which a 10% 
sample (n=10) is to be selected. The proposals are then arranged in order of 
decreasing bid price and numbered accordingly; that is, the proposal with the largest 
bid price is number 1, the second largest number 2. and so on. The proposals are now 
ready to be separated into 10 different baskets. Starting with proposals 1 through 10 
t those with the largest bid prices), one proposal is placed in each basket. Each 

Mt is realized there may be other relevant factors besides bid price that should be 
considered in the definition of "representative". For the purposes of this paper, 
however, it will suffice to say that sophisticated software can quickly balance baskets 
for type of work, degree of labor intensity, level of technology, etc. In short, whatever 
characteristics are identified as potentially important to the value of an audit will be 
"balanced" by the basket method where possible. 
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successive group of 10 proposals are assigned, one-per-basket, using the following rule: 
the hugest unassigned proposal is placed in the basket with the smallest sum of bid prices. 
For the second group of 10 proposals, this rule results in pairing proposal 11 with 10, 
12 with 9, . . ., and 20 with 1. Basket subtotals are then calculated and the assignment 
rule applied to the third group of 10 proposals. This is repeated until all the proposals 
have been assigned. [Ref. 1: pg.10] 

Due to the balancing of basket totals at each stage of the basket assignment 
process, the resulting assignment should result in nearly equal basket totals. Should 
additional balancing be required, the previously mentioned computer program can be 
used (via a swapping algorithm) to bring basket totals into closer agreement. 

2. Estimating Negotiated Prices for Unsampled Proposals 

After the baskets are formed, one is selected at random and all its proposals 
are audited and negotiated. Using the results of the sample negotiation, the sample 
ratio factor. R. is computed as in equation 2.1. 

R = Total negotiated price of sample Total bid price of sample (eqn 2.1) 

The totai proposal value of the population is then multiplied by the sample ratio 
factor, R, to determine the population audit result. This value will be the estimated 
true value of the population. 2 



2 The sample ratio factor could also be applied individually to each unsampled 
proposal, the values summed and the total added to the sum of the negotiated values 
of sampled proposals. The result would be the same. 
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